<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
 
 <title>papy - Google Code</title>
 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
 
 <link rel="shortcut icon" href="http://www.gstatic.com/codesite/ph/images/defaultlogo.png"/>
 <link type="text/css" rel="stylesheet"
       href="./_static/custom.css" />
 <link type="text/css" rel="stylesheet"
       href="http://www.gstatic.com/codesite/ph/3770121226765569314/css/ph_core.css" />
 <link type="text/css" rel="stylesheet"
       href="http://www.gstatic.com/codesite/ph/3770121226765569314/css/ph_detail.css" />
 <link rel="stylesheet"  type="text/css"
       href="./_static/pygments.css" /> 
 
 <link type="application/atom+xml" rel="alternate"
 href="/feeds/p/papy/svnchanges/basic?path=/wiki/" />
 
 
<!--[if IE]>
 <link type="text/css" rel="stylesheet"
       href="http://code.google.com/hosting/css/d_ie.css" />
<![endif]-->

<script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
          URL_ROOT:    '',
          VERSION:     '1.0b1',
          COLLAPSE_MODINDEX: false,
          FILE_SUFFIX: '.html'
      };
    </script>
    <script type="text/javascript" src="./_static/jquery.js"></script>
    <script type="text/javascript" src="./_static/interface.js"></script>
    <script type="text/javascript" src="./_static/doctools.js"></script>
    <link rel="contents" title="Global table of contents" href="contents.html" />
    <link rel="index" title="Global index" href="genindex.html" />
    <link rel="search" title="Search" href="search.html" />
    <link rel="top" title="papy v1.0b1 documentation" href="index.html" />
    <link rel="next" title="Tutorials" href="tutorials.html" />
    <link rel="prev" title="About the Parallelism in PaPy" href="parallel.html" />
</head>
<body class="t6">
 <div id=gaia>
  <font size="-1">
 
 <a href="http://code.google.com/p/support/wiki/WhatsNew" style="color:#a03">What's new?</a>
 | <a href="http://code.google.com/p/support/">Help</a>
 
 </font> 

 </div>
 <table style="padding:0px; margin: 0px 0px -6px 0; width:100%" cellpadding=0 cellspacing=0>
 <tr>
 <td><a href="/"><img src="http://www.gstatic.com/codesite/ph/images/defaultlogo.png" alt=papy/></a></td>
 <td>
 
 <div id="pname">
 <a href="http://code.google.com/p/papy/" style="text-decoration:none; color:#000">papy</a>
 
 </div>
 <div id="psum">
 <i><a href="http://code.google.com/p/papy/" style="text-decoration:none; color:#000">papy v1.0b1 documentation</a></i>
 </div>
 
 </td>
 <td style="white-space:nowrap; text-align:right">
 
 <form action="http://code.google.com/hosting/search">
 <input size=30 name=q value=""/>
 <input type=submit name=projectsearch value="Search Projects" />
 <input type=submit name=websearch value="Search the Web" />
 </form>
 
 </tr>
 </table>


<table id="mt" cellspacing="0" cellpadding="0" width="100%" border="0">
 <tr>
 <th onclick="if (!cancelBubble) _go('http://code.google.com/p/papy/');">
 <div class="tab inactive">
 <div class="round4"></div>
 <div class="round2"></div>
 <div class="round1"></div>
 <div class="box-inner">
 <a onclick="cancelBubble=true;" href="http://code.google.com/p/papy/">Project&nbsp;Home</a>
 </div>
 </div>
 </th><td>&nbsp;&nbsp;</td>
 
 
 
 
 <th onclick="if (!cancelBubble) _go('http://code.google.com/p/papy/downloads/list');">
 <div class="tab inactive">
 <div class="round4"></div>
 <div class="round2"></div>
 <div class="round1"></div>
 <div class="box-inner">
 <a onclick="cancelBubble=true;" href="http://code.google.com/p/papy/downloads/list">Downloads</a>
 </div>
 </div>
 </th><td>&nbsp;&nbsp;</td>
 
 
 
 
 
 <th onclick="if (!cancelBubble) _go('http://code.google.com/p/papy/w/list');">
 <div class="tab inactive">
 <div class="round4"></div>
 <div class="round2"></div>
 <div class="round1"></div>
 <div class="box-inner">
 <a onclick="cancelBubble=true;" href="http://code.google.com/p/papy/w/list">Wiki</a>
 </div>
 </div>
 </th><td>&nbsp;&nbsp;</td>
 
 
 
 
 
 
 
 <th onclick="if (!cancelBubble) _go('http://code.google.com/p/papy/issues/list');">
 <div class="tab inactive">
 <div class="round4"></div>
 <div class="round2"></div>
 <div class="round1"></div>
 <div class="box-inner">
 <a onclick="cancelBubble=true;" href="http://code.google.com/p/papy/issues/list">Issues</a>
 </div>
 </div>
 </th><td>&nbsp;&nbsp;</td>
 
 
 
 
 
 
 <th onclick="if (!cancelBubble) _go('http://code.google.com/p/papy/source/list');">
 <div class="tab inactive">
 <div class="round4"></div>
 <div class="round2"></div>
 <div class="round1"></div>
 <div class="box-inner">
 <a onclick="cancelBubble=true;" href="http://code.google.com/p/papy/source/list">Source</a>
 </div>
 </div>
 </th><td>&nbsp;&nbsp;</td>
 
 
 
 <td width="100%">&nbsp;</td>
 </tr>
</table>
<table cellspacing="0" cellpadding="0" width="100%" align="center" border="0" class="st">
 <tr>
 
 
 
 <td>
 <div class="issueList">
<div class=isf>
 
 
 
 <a href="index.html">papy v1.0b1 documentation</a> |
 <span>
   <a href="genindex.html" title="General Index"
      accesskey="I">index</a>
 </span> |
 <span>
   <a href="modindex.html" title="Global Module Index"
      accesskey="M">modules</a>
 </span> |
 <span>
   <a href="tutorials.html" title="Tutorials"
      accesskey="N">next</a>
 </span> |
 <span>
   <a href="parallel.html" title="About the Parallelism in PaPy"
      accesskey="P">previous</a>
 </span> |
 <span class="inIssueList">
   <span>Search</span>
    <form action="search.html" method="GET" style="display:inline">
      <input size=30 name=q value=""/>
      <input type=submit name=projectsearch value="Search Documentation" />
      <input type="hidden" name="check_keywords" value="yes" />
      <input type="hidden" name="area" value="default" />
    </form>
 </span>

</div>
</div>

 </td>
 
 
 
 
 
 
 
 
 <td height=4 align=right valign=top class="bevel-right">
 <div class="round4"></div>
 <div class="round2"></div>
 <div class="round1"></div>
 </td>
 </tr>
</table>
<script>
 var cancelBubble = false;
 function _go(url) { document.location = url; }
</script>

<div id="maincol">
<!-- IE -->



<style>
 #downloadbox { 
 padding: 6px; 
 overflow: hidden;
 white-space: nowrap;
 text-overflow: ellipsis;
 }
 #downloadbox a {
 margin: 0 0 0 0;
 display: block;
 padding-left: 0px;
 }
 #owners a, #members a { white-space: nowrap; }
</style>

<div style="float:right; width:25em; margin:0 0 1em 2em">

 
 <div class=pmeta_bubble_bg>
 <div class="round4"></div>
 <div class="round2"></div>

 <div class="round1"></div>
 <div class="box-inner">
 <table class="pmeta" cellpadding=5>
 <tr><th>Previous:</th>
 <td><ul>
 
 <li>
   <a href="parallel.html" title="previous chapter">About the Parallelism in <em>PaPy</em></a>
 </li>
 

 
 </ul></td>
 </tr>
 <tr><th>Next:</th>
 <td><ul>
 
 <li>
   <a href="tutorials.html" title="next chapter">Tutorials</a>
 </li>
 

 
 </ul></td>
 </tr>
 
 
 
 </table>

 </div>
 <div class="round1"></div>
 <div class="round2"></div>
 <div class="round4"></div>
 </div>
 
 
 
 <div style="background:#ddf8cc; margin-bottom: 5px;table-layout:fixed">

 <div class="round4"></div>

 <div class="round2"></div>
 <div class="round1"></div>
 <div class="box-inner">
 <div id=downloadbox>
  <b>Table of Contents</b><br/>
  <table class="pmeta toc" cellpadding=5><tr><td>
	<ul>
<li><a class="reference external" href="">Inter Process Communication</a><ul>
<li><a class="reference external" href="#the-unit-of-execution">The unit of execution</a></li>
<li><a class="reference external" href="#a-simple-example">A simple example</a></li>
<li><a class="reference external" href="#how-does-it-work">How does it work?</a></li>
<li><a class="reference external" href="#how-do-database-hooks-work">How do database &#8220;hooks&#8221; work?</a></li>
<li><a class="reference external" href="#which-method-should-i-choose">Which method should I choose?</a></li>
</ul>
</li>
</ul>

  </td></tr></table>

 
 </div>
 </div>
 <div class="round1"></div>
 <div class="round2"></div>
 <div class="round4"></div>
 </div>
 
 


 </div>
</div>

 <div style="padding:0 3em 1.2em 0">
   
     
  <div class="section" id="inter-process-communication">
<h1>Inter Process Communication<a class="headerlink" href="#inter-process-communication" title="Permalink to this headline">¶</a></h1>
<p>This chapter deals with the details of inter-process communication (IPC) in
<em>PaPy</em>. By it&#8217;s design <em>PaPy</em> is a rather high-level engine for the execution of
chained tasks and the details of inter-process communication are by default
hidden from the user. This does not mean that it is not possible to influence
how processes are communicated i.e. synchronize and exchange data. Optimization
is done at the cost of generality e.g. knowing that two processes will execute
on a shared memory UNIX system allows to communicate them via pipes (FIFOs),
which will skip some computation and eliminate a possible bottleneck. This
section should be read together with the API documentation for
<tt class="docutils literal"><span class="pre">papy.workers.io</span></tt>.</p>
<div class="section" id="the-unit-of-execution">
<h2>The unit of execution<a class="headerlink" href="#the-unit-of-execution" title="Permalink to this headline">¶</a></h2>
<p>A <em>Piper</em> object is assigned to an <em>IMap</em> instance which uses a process/thread
pool to parallelize the execution. All functions used to create a <em>Worker</em> are
called in a single call and no IPC is necessary. <em>Pipers</em> on the other hand
might be executed in different processes and on different machines running
different operating systems. By default <em>Pipers</em> are connected by a pair of
locked pipe objects via a &#8220;manager&#8221; process, which is the process used to
execute/start the pipeline.</p>
</div>
<div class="section" id="a-simple-example">
<h2>A simple example<a class="headerlink" href="#a-simple-example" title="Permalink to this headline">¶</a></h2>
<p>Using a &#8220;manager&#8221; process to communicate two other process is inefficient as it
involves two passes of pickling/unpickling and creates a potential bottleneck as
half of all computations is done by the intermediate process. The solution is to
either a) bypass the double pipe connection and connect processes directly, b)
use a more efficient serialization protocol. Both methods involve simply adding
dumping(output) and loading(input) worker-functions to the <em>Workers</em> i.e.:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">papy</span> <span class="kn">import</span> <span class="n">workers</span>
<span class="n">upstream</span> <span class="o">=</span> <span class="n">Worker</span><span class="p">((</span><span class="n">func</span><span class="p">,</span> <span class="n">workers</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">pickle_dumps</span><span class="p">,</span> <span class="n">workers</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">dump_item</span><span class="p">),</span>\
               <span class="p">((),</span>    <span class="p">(),</span>                      <span class="p">(</span><span class="s">&#39;tcp&#39;</span><span class="p">)))</span>
<span class="n">downstram</span> <span class="o">=</span> <span class="n">Worker</span><span class="p">((</span><span class="n">workers</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">load_item</span><span class="p">,</span> <span class="n">workers</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">pickle_loads</span><span class="p">,</span> <span class="n">func</span><span class="p">),</span>\
                <span class="p">((),</span>                    <span class="p">(),</span>                      <span class="p">()))</span>
<span class="n">up_piper</span> <span class="o">=</span> <span class="n">Piper</span><span class="p">(</span><span class="n">upstream</span><span class="p">,</span> <span class="n">parallel</span> <span class="o">=</span><span class="n">some_IMap_instance</span><span class="p">)</span>
<span class="n">down_piper</span> <span class="o">=</span> <span class="n">Piper</span><span class="p">(</span><span class="n">downstram</span><span class="p">,</span> <span class="n">parallel</span> <span class="o">=</span><span class="n">some_different_IMap_instance</span><span class="p">)</span>
<span class="n">pipes</span> <span class="o">=</span> <span class="n">Plumber</span><span class="p">()</span>
<span class="n">pipes</span><span class="o">.</span><span class="n">add_pipe</span><span class="p">((</span><span class="n">up_piper</span><span class="p">,</span> <span class="n">down_piper</span><span class="p">))</span>
</pre></div>
</div>
<p>In this example we created two <em>Worker</em> instances, which are used to create
<em>Piper</em> instances connected within a <em>Dagger</em> instance and executed by
different processes possibly on different physical machines. Because of this a
networked method of communication has been chosen &#8216;tcp&#8217;. This method involves
sending data over a network socket. Data has to be serialized before it can be
passed to another process. Within <tt class="docutils literal"><span class="pre">multiprocessing</span></tt> this is done via pickling,
<em>RPyC</em> uses an internal protocol called brine. <em>PaPy</em> has built-in workers
which support the pickle, json and marshall protocols. Pickle is the most
general protocols and most Python objects can be pickled. Currently Json and
marshall might be faster, but they have limitations (compability between Python
versions and the range of serializable Python objects). The <tt class="docutils literal"><span class="pre">load_item</span></tt> worker
will auto discover the type of communication. The currently supported methods of
communication are &#8216;tcp&#8217; or &#8216;udp&#8217; for network communications, &#8216;fifo&#8217; or &#8216;shm&#8217; to
communicate processes on the same physical machine and hooks to databases
currently &#8216;sqlite&#8217; and &#8216;mysql&#8217;. Data can also be exchanged via temporary files
&#8216;file&#8217;. Temporary files can be used to communicate remote hosts if they are
accessible from both e.g. via NFS or Samba.</p>
</div>
<div class="section" id="how-does-it-work">
<h2>How does it work?<a class="headerlink" href="#how-does-it-work" title="Permalink to this headline">¶</a></h2>
<p>If the user decides to use custom communication methods the inefficient
double-pipe connection is used to transfer only a very small amount of data and
effectively to synchronize the processes. For example for TCP based
communication it is the hostname and port and the type of the protocol &#8216;tcp&#8217;.
For file, shm and FIFO based communication it is just the file name. This amount
of data will not be a bottleneck for pipelines of any size.</p>
</div>
<div class="section" id="how-do-database-hooks-work">
<h2>How do database &#8220;hooks&#8221; work?<a class="headerlink" href="#how-do-database-hooks-work" title="Permalink to this headline">¶</a></h2>
<p>The <em>PaPy</em> <tt class="docutils literal"><span class="pre">dump_db_item</span></tt> and <tt class="docutils literal"><span class="pre">load_db_item</span></tt> worker functions allow to
communicate Python processes via a database. The data in the database can be
stored only until it is retrieved or persistantly and serve as a way to
check-point the pipeline. How the data is stored in a database depends on the
type of the database, currently &#8216;mysql&#8217; and &#8216;sqlite&#8217; databases are supported.
Sqlite database files should not be shared over NFS, but can be written and read
by different processes.:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">papy</span> <span class="kn">import</span> <span class="n">workers</span>
<span class="n">upstream</span> <span class="o">=</span> <span class="n">Worker</span><span class="p">((</span><span class="n">func</span><span class="p">,</span> <span class="n">workers</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">pickle_dumps</span><span class="p">,</span> <span class="n">workers</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">dump_db_item</span><span class="p">),</span>\
                 <span class="p">((),</span>    <span class="p">(),</span>                      <span class="p">(</span><span class="s">&#39;sqlite&#39;</span><span class="p">)))</span>
<span class="n">downstram</span> <span class="o">=</span> <span class="n">Worker</span><span class="p">((</span><span class="n">workers</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">load_db_item</span><span class="p">,</span> <span class="n">workers</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">pickle_loads</span><span class="p">,</span> <span class="n">func</span><span class="p">),</span>\
                  <span class="p">((),</span>                    <span class="p">(),</span>                         <span class="p">()))</span>
<span class="n">up_piper</span> <span class="o">=</span> <span class="n">Piper</span><span class="p">(</span><span class="n">upstream</span><span class="p">,</span> <span class="n">parallel</span> <span class="o">=</span><span class="n">some_IMap_instance</span><span class="p">)</span>
<span class="n">down_piper</span> <span class="o">=</span> <span class="n">Piper</span><span class="p">(</span><span class="n">downstram</span><span class="p">,</span> <span class="n">parallel</span> <span class="o">=</span><span class="n">some_different_IMap_instance</span><span class="p">)</span>
<span class="n">pipes</span> <span class="o">=</span> <span class="n">Plumber</span><span class="p">()</span>
<span class="n">pipes</span><span class="o">.</span><span class="n">add_pipe</span><span class="p">((</span><span class="n">up_piper</span><span class="p">,</span> <span class="n">down_piper</span><span class="p">))</span>
</pre></div>
</div>
</div>
<div class="section" id="which-method-should-i-choose">
<h2>Which method should I choose?<a class="headerlink" href="#which-method-should-i-choose" title="Permalink to this headline">¶</a></h2>
<p>The recommended method of communication depends whether the processes run on the
same or different physical machines, what the operating systems of the Python
processes and the size of the exchanged data. Small lists, objects, strings,
etc. (the size of up to hundreds of kB after serialization) should be
transferred using the default method i.e. without using the dump_item worker.
Only for large objects non-standard communication methods should be considered.
If the processes run on a shared memory UNIX system you should use FIFOs (pipes)
or shared memory (this requires the posix_ipx module). Using FIFOs on Windows
systems is currently not supported (but might be in future) and Windows is not
POSIX compliant so you are left with files, which might be fast enough for
typical applications. Files on a network share are also the recommended method
to communicate Windows-based processes. Networking i.e. &#8216;tcp&#8217; and &#8216;udp&#8217; requires
forking of the process, which evaluates the <tt class="docutils literal"><span class="pre">dump_item</span></tt> function.  Forking is
not supported on Windows, but should work well on all UNIX systems. UDP should
provide higher performance than TCP, but should only be used on reliable,
collision-less networks. When using UDP you are not guaranteed that all data
will be transmitted over the network, this will yield WorkerErrors, which in
turn will require you to re-run the pipeline for failed input items.</p>
</div>
</div>


   
 
 
 </div>
<div id="footer" dir="ltr">
 
 <div class="text">
 
 &copy;2008 Google -
 <a href="http://code.google.com/">Code Home</a> -
 <a href="http://code.google.com/tos.html">Terms of Service</a> -
 <a href="http://www.google.com/privacy.html">Privacy Policy</a> -
  
 </body>
</html>
